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(54) Vehicle surroundings monitoring apparatus 



(57) A vehicle surroundings monitoring apparatus 2 
has cameras having optical axes respectively, provided 
with imaging lens systems 5,6 each having a fish-eye 
lens 5a, 6a, image processing units 7, 8 and a calculat- 
ing unit 9. The image processing units 7, 8 detect mov- 
ing objects, such as vehicles traveling alongside a host 
vehicle provided with the vehicle surroundings monitor- 
ing apparatus, and oncoming vehicles, on the basis of 
optical flow between two images taken by the cameras 



at different times, respectively, and set a representative 
point representing the detected moving object on the 
surface of a road. The calculating unit calculates the po- 
sition of the moving object relative to the host vehicle on 
the basis of displacement angles a and p of the repre- 
sentative point A with respect to the optical axis on an 
assumption that the representative point A is on a plane 
including a surface with which the host vehicle is in con- 
tact. 
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Description 

BACKGROUND OF THE INVENTION 

Field of the Invention 5 

[0001] The present invention relates to a vehicle sur- 
roundings monitoring apparatus for a vehicle, capable 
of detecting moving objects, such as vehicles traveling 
around a vehicle provided with the vehicle surroundings 10 
monitoring apparatus (hereinafter, referred to as "host 
vehicle"), on the basis of images provided by a fish-eye 
lens. 

Description of the Related Art 15 

[0002] Various types of vehicle surroundings monitor- 
ing apparatuses have been proposed and applied to 
practical uses in recent years for the improvement of 
safety of vehicles. Those vehicle surroundings monitor- 20 
ing apparatuses are capable of three-dimensionally rec- 
ognizing roads and traffic conditions in a practically sat- 
isfactory accuracy and in a practically permissible time 
by obtaining images of scenery extending ahead of the 
host vehicle or objects lying in front of the host vehicle 25 
by a camera and processing the images. 
[0003] A technique proposed in, for example, JP-A 
No. 283461/1998 by the applicant of the present patent 
application subjects images taken by a pair of cameras 
(stereo camera) to a stereoscopic process to detect the 30 
preceding vehicle and obstacles. 
[0004] It is desirable, for the further improvement of 
the safety of the host vehicle, to give information about 
other vehicles and the like traveling beside the host ve- 
hicle to the driver in addition to that about a space in 35 
front of the host vehicle. Preferably, an image of scenery 
extending around the side of the host vehicle is obtained 
by a camera provided with a fish-eye lens to acquire in- 
formation about a wide range of the side scenery. For 
example, a technique disclosed in JP^A No. 40 
202180/1997 obtains an image of the side scenery by 
a fish-eye lens camera, converts the image into a regu- 
lar image signal by a coordinate transformation circuit, 
and displays an optional region of a rectified image on 
the screen of a monitor. However, since this technique *5 
merely displays the image of a predetermined part of 
the vehicle's surroundings on the screen of the monitor, 
it is difficult to give the driver distance information rep- 
resenting distances between the host vehicle and other 
vehicles traveling beside the host vehicle. 50 
[0005] To deal with such a difficulty in giving the driver 
distance information, a technique disclosed in, for ex- 
ample, JP-B No. 2611173 uses a monitoring system in- 
cluding a plurality of cameras respectively provided with 
fish-eye lenses. The monitoring system calculates 55 
three-dimensional coordinates indicating the three-di- 
mensional position of a moving object on the basis of a 
plurality of wide-angle images provided by the plurality 



of cameras to acquire distance information about the 
distance between the host vehicle and a vehicle 
traveling beside the host vehicle. However, it is not eco- 
nomically preferable to use such a monitoring system of 
complicated construction for obtaining information 
about a side space extending around the side of the host 
vehicle, which may be rough as compared with informa- 
tion about a forward space extending in front of the host 
vehicle. 

SUMMARY OF THE INVENTION 

[0006] The present invention has been made in view 
of such circumstances, and it is therefore an object of 
the present invention to provide a vehicle surroundings 
monitoring apparatus of simple construction capable of 
obtaining proper information about the surroundings of 
a host vehicle. 

[0007] According to one aspect of the present inven- 
tion, a vehicle surroundings monitoring apparatus for 
monitoring the surroundings of a host vehicle includes: 
image obtaining means each having an imaging lens 
system including a fish-eye lens; an image processing 
means capable of detecting a moving object on the ba- 
sis of an optical flow between two images obtained by 
the image obtaining means respectively at different 
times, by using one of the images as a reference image, 
and of setting a representative point representing the 
moving object on the surface of a road; and a calculating 
means for calculating the position of the moving object 
relative to the vehicle on the basis of a displacement 
angle of the representative point from the optical axis 
on an image. 

[0008] In the vehicle surroundings monitoring appa- 
ratus according to the present invention, the image ob- 
taining means are disposed on right and left side parts 
of a body included in the host vehicle. 
[0009] In the vehicle surroundings monitoring appa- 
ratus according to the present invention, the imaging 
lens system has an optical axis inclined slightly rearward 
to a direction parallel to the width of the vehicle. 
[0010] In the vehicle surroundings monitoring appa- 
ratus according to the present invention may further in- 
clude a display means for displaying the positions of 
moving objects relative to the host vehicle, capable of 
indicating the velocities of the moving objects relative to 
the host vehicle by vector notation. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] By way of example only, specific embodiments 
of the present invention will now be described, with ref- 
erence to the accompanying drawings, in which: 

Fig. 1 is a functional block diagram of a vehicle sur- 
roundings monitoring apparatus in a preferred em- 
bodiment according to the present invention; 
Fig. 2 is a diagrammatic view for explaining image 
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regions obtained by a image obtaining means in an 
example; 

Fig. 3 is a simplified diagram showing a distribution 
of velocity vectors; 

Fig. 4 is a diagram showing a coordinate system for 
indicating angle of displacement from an optical ax- 
is; 

Fig. 5 is a diagrammatic view for explaining the po- 
sitional relation between a vehicle and a represent- 
ative point; 

Fig. 6 is a diagrammatic view of for explaining a 
method of calculating the distance between a vehi- 
cle and a representative point with respect to a di- 
rection parallel to an optical axis; 
Fig. 7 is a diagrammatic view showing an image dis- 
played on the screen of a display; and 
Fig. 8 is a diagrammatic view for explaining image 
regions obtained by a image obtaining means in an- 
other example. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0012] Referring to Fig. 1, a vehicle provided with a 
vehicle surroundings monitoring apparatus 2 in a pre- 
ferred embodiment according to the present invention, 
i.e. a host vehicle, has a body 1. The vehicle surround- 
ings monitoring apparatus 2 obtains images of outside 
sceneries in set ranges, recognizes outside moving ob- 
jects, such as vehicles traveling alongside the host ve- 
hicle and oncoming vehicles, and monitors the outside 
moving objects. The vehicle surroundings monitoring 
apparatus 2 comprises cameras (image obtaining 
means) 5 and 6 disposed on the right and the left side 
of the body 1 respectively, image processing units (im- 
age processing means) 7 and 8 capable of processing 
images obtained by the cameras 5 and 6 to detect mov- 
ing objects, such as vehicles traveling alongside the 
host vehicle and oncoming vehicles, and setting repre- 
sentative points representing the detected moving ob- 
jects on the surface of a road, a calculating unit (calcu- 
lating means) 9 capable of calculating the respective po- 
sitions of the moving objects relative to the host vehicle 
on the basis of angles of displacement of the represent- 
ative points on the image to an optical axis, and a display 
(display means) 1 0 for displaying an image representing 
the calculated positional relation between the host ve- 
hicle and the moving objects. 

[0013] The cameras 5 and 6 comprise imaging lens 
systems 5a and 6a including fish-eye lenses, and solid- 
state image sensors, not shown, respectively. The im- 
aging lens systems 5a and 6a focus incident light on the 
surfaces of the solid-state image sensors, respectively. 
The cameras 5 and 6 have a large angular range of 1 80 
degrees. As shown in Figs . 1 and 2, the cameras 5 and 
6 are attached, for example, to rear parts of the body 1 
at a height H, respectively, such that the optical axes O 
of the imaging lens systems 5a and 6a are inclined 
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slightly rearward to a direction parallel to the width of 
the body 1 in a horizontal plane. As shown in Fig. 2, the 
respective angular ranges of the cameras 5 and 6 cover 
substantially entirely a space around the right and the 

5 left side of the body 1 and behind the body 1. In this 
embodiment, the vehicle is supposed to be a wagon and 
the body 1 is of a wagon type. Desirably, the cameras 
are attached to the D-pillars of the body 1. 
[0014] The cameras 5 and 6 are connected to the im- 

10 age processing units 7 and 8 to give images to the image 
processing units 7 and 8, respectively. Also connected 
to the image processing units 7 and 8 are speed sensor 
11 for measuring the traveling speed of the host vehicle 
and a steering angle sensor 12 for measuring steering 

15 angles. Each of the image processing units 7 and 8 pro- 
duces an optical flow from the two successive images 
obtained at different times by each of the cameras 5 and 
6 to detect moving objects, such as vehicles traveling 
alongside the host vehicle, and oncoming vehicles. That 

20 is, each of the image processing units 7 and 8 produces 
an optical flow indicating a distribution of velocity vectors 
in each of predetermined pixel blocks through the 
matching of the luminance pattern of the pixel block in 
the image taken at the later time, i.e., a reference image, 

25 and that of the pixel block in the image taken at the ear- 
lier time, i.e., a comparative image. 
[0015] As shown in Fig. 3, velocity vectors represent- 
ing the velocities of fixed objects, such as the surface 
of a road, have a direction substantially opposite to the 

30 traveling direction of the host vehicle and a magnitude 
substantially equal to the traveling velocity of the host 
vehicle. In contrast, velocity vectors representing the ve- 
locities of moving objects, such as vehicles traveling 
alongside the host vehicle, and oncoming vehicles, 

35 have magnitudes different from that of a velocity vector 
representing the traveling speed of the host vehicle. The 
image processing units 7 and 8 recognize a group of 
similar velocity vectors having magnitudes different 
from that of the velocity vector of the host vehicle as a 

40 velocity vector group representing the velocity of a sin- 
gle moving object for moving object detection. 
[0016] The image processing units 7 and 8 set a rep- 
resentative point on a road surface on the basis of the 
detected velocity of the moving object for the moving 

45 object. That is, the image processing units 7 and 8 set 
a position, where the velocity vector of the lowest point 
of the moving object is in contact with the velocity vector 
of the fixed objects, as a contact point on the road sur- 
face. As shown in Fig. 3, when the moving object is a 

50 vehicle traveling alongside the host vehicle, the contact 
point where the moving object is in contact with the sur- 
face of the ground corresponds substantially to a point 
where a tire of the vehicle is in contact with the surface 
of the ground. The image processing units 7 and 8 draw 

55 a line segment L1 passing the contact point and parallel 
to the velocity vector group of the moving object on the 
image, draw a line segment L2 extending between a ve- 
locity vector at the head or the tail of the velocity vector 
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group and nearest to the host vehicle, and the center of 
field (optical axis O), and set a representative point A at 
the intersection of the line segments L1 and L2. As 
shown in Fig. 3, the representative point A does not nec- 
essarily coincide with an end of the body (moving ob- 
ject). In such a case, the representative point A has an 
error on the side of the host vehicle (safety side). 
[0017] The image processing units 7 and 8 are con- 
nected to the calculating unit 9 to give information about 
the representative point A of each of moving objects on 
the image. The calculating unit 9 calculates the vertical 
displacement angle a and horizontal displacement an- 
gle p of the representative point A with respect to the 
optical axis O on the image. A coordinate system for in- 
dicating displacement angles with respect to the optical 
axis O on the image is set in the calculating unit 9, and 
the calculating unit 9 calculates the vertical displace- 
ment angle a and horizontal displacement angle p of the 
representative point A on the basis of the coordinate 
system. Then, the calculating unit 9 calculates a dis- 
tance D1 to the representative point A along the optical 
axis O, and a distance D2 from the representative point 
A to the optical axis O along a perpendicular from the 
representative point A to the optical axis O. 
[0018] Since the optical axis O extends in a horizontal 
plane at a height H from the surface of the road as shown 
in Fig. 6, the distance D1 is expressed by Expression 
(1) on an assumption that the surface of the road on 
which the representative point A is located and a contact 
surface with which the host vehicle is in contact are in- 
cluded in the same plane. 

D1 = H tan(90- a) (1) 

[0019] D2 is expressed by Expression (2). 

D2 = D1-tanp (2) 

Thus, the position of the moving object relative to that 
of the host vehicle is defined. 

[0020] If a part of the body 1 is included in the field of 
view of the imaging lens systems 5a and 6a as shown 
in Fig. 6, the displacement angle cc2 of a boundary be- 
tween the body 1 and the surface of the road may be 
measured and may be used as reference information 
about the displacement angle a. The displacement an- 
gle cc2 is dependent on the positions of the cameras 5 
and 6, and the morphology of the body 1 . If the positions 
of the cameras 5 and 6 include an error, the displace- 
ment angle a2 changes. Therefore, the accuracy of de- 
tection of the position of the moving object relative to 
that of the host vehicle can be improved by correcting 
the displacement angle a according to an error in the 
displacement angle a2. 

[0021 ] The calculating unit 9 generates an output sig- 
nal to be given to the display 10 on the basis of informa- 



tion about the respective positions of the moving objects 
relative to the position of the host vehicle. For example, 
a triangle representing the host vehicle is displayed in 
a central part of the screen of the display 10, and rec- 

5 tangles representing the moving objects, such as vehi- 
cles traveling alongside the host vehicle and oncoming 
vehicles, are displayed on the screen of the display 10. 
The velocities of the moving objects relative to the host 
vehicle are displayed by vector notation on the screen 

10 of the display 10. When the distance between the host 
vehicle and the moving object decreases below a pre- 
determined threshold, a warning may be given to that 
effect by, for example, flickering the rectangle represent- 
ing the moving object. 

15 [0022] Since the vehicle surroundings monitoring ap- 
paratus 2 in this embodiment obtains images of objects 
around the host vehicle by the cameras 5 and 6 having 
the imaging lens systems 5a and 6a provided with fish- 
eye lenses, respectively, the vehicle surroundings mon- 

20 itoring apparatus 2 is capable of Obtaining images of a 
wide range by using a small number of image obtaining 
devices. 

[0023] The vehicle surroundings monitoring appara- 
tus 2 is readily able to detect moving objects, such as 
25 vehicles traveling alongside the host vehicle, and on- 
coming vehicles, by producing an optical flow between 
two images successively obtained at different times, re- 
spectively. 

[0024] The position of a moving object relative to the 

30 host vehicle can be readily calculated on the basis of 
the displacement angle from the optical axis O by setting 
the representative point A representing the detected 
moving object on the surface of the road. That is, the 
cameras 5 and 6 are set on the host vehicle with their 

35 optical axes O extended in a horizontal plane, and the 
distance between the host vehicle and the representa- 
tive point A can be readily calculated on the basis of the 
displacement angle from the optical axis O on an as- 
sumption that the representative point A is in a plane 

40 including the surface with which the host vehicle is in 
contact. Therefore, any stereoscopic image processing 
operation and the like are unnecessary, the positional 
relation between the host vehicle and the moving object 
can be recognized from an image obtained by the single 

45 camera 5 or 6. Thus, proper information about the am- 
bience of the host vehicle can be obtained by the vehicle 
surroundings monitoring apparatus of a simple configu- 
ration. The surroundings of the host vehicle can be ef- 
fectively monitored by a small number of imaging devic- 

50. es by disposing the cameras 5 and 6 on the right and 
the left side of the body 1, respectively. 
[0025] Images of regions which are difficult for the 
driver to monitor can be effectively obtained by dispos- 
ing the cameras 5 and 6 with their optical axes O inclined 

55 slightly rearward to a direction parallel to the width of 
the host vehicle. 

[0026] Indication of the velocities of the moving ob- 
jects relative to the host vehicle by vector notation on 
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the display 10 is effective in making the driver recognize 
vehicles traveling alongside the host vehicle, and on- 
coming vehicles. 

[0027] Although the vehicle surroundings monitoring 
apparatus 2 in the foregoing embodiment is provided 
with the cameras 5 and 6 disposed at the right and the 
left part of the body 1, the vehicle surroundings moni- 
toring apparatus may be provided with cameras 15 and 
16 disposed at right and left parts of the body 1, respec- 
tively, and a camera 17 disposed at a rear part of the 
body 1 as shown in Fig. 8. When the vehicle surround- 
ings monitoring apparatus is provided with the cameras 
15, 16 and 17, it is desirable to dispose the cameras 15 
and 16 on the body 1 with their optical axes O extended 
in a middle part of the body 1 in parallel to the width of 
the body 1 . 

[0028] As apparent from the foregoing description, 
according to the present invention, the vehicle surround- 
ings monitoring apparatus is simple in construction and 
is capable of acquiring information about the surround- 
ings of the vehicle by using a small number of image 
obtaining means. 

[0029] Although the invention has been described in 
its preferred embodiment with a certain degree of par- 
ticularity, obviously many changes and variations are 
possible therein. It is therefore to be understood that the 
present invention may be practiced otherwise than as 
specifically described herein without departing from the 
scope thereof. 
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cording to claim 2, wherein each imaging lens sys- 
tem has an optical axis inclined slightly rearward to 
a direction parallel to the width of the host vehicle. 

The vehicle surroundings monitoring apparatus ac- 
cording to any one of claims 1 to 3 further compris- 
ing a display means for displaying positions of mov- 
ing objects relative to that of the host vehicle, capa- 
ble of indicating velocities of the moving objects rel- 
ative to the host vehicle by vector notation. 

A vehicle comprising the vehicle surroundings mon- 
itoring apparatus as claimed in any of claims 1 to 4. 



1. A vehicle surroundings monitoring apparatus com- 
prising: 

image obtaining means each having an imag- 
ing lens system including a fish-eye lens; 
an image processing means capable of detect- 
ing a moving object on the basis of an optical 
flow between two images obtained by the im- 
age obtaining means respectively at different 
times, using one of the images as a reference 
image, and of setting a representative point 
representing the moving object on a surface of 
a road; and 

a calculating means for calculating the position 
of the moving object relative to the vehicle on 
the basis of a displacement angle of the repre- 
sentative point from the optical axis on an im- 
age. 

2. The vehicle surroundings monitoring apparatus ac- 
cording to claim 1, wherein the image obtaining 
means are disposed on right and left side parts of 
a body included in a host vehicle, respectively. 
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3. The vehicle surroundings monitoring apparatus ac- 
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